1. Field of Invention
This invention relates to image analysis, specifically to identifying friendly and enemy units in a soldier's head-mounted display.
2. Discussion of the Background
In September 2000, the U.S. Army performed the first operational test of the Land Warrior soldier system. The system connects soldiers on the battlefield via a wireless network. In addition, each Land Warrior carries a computer connected to a GPS receiver, scope video camera, and head-mounted display. The head-mounted display presents video from the scope camera attached to the soldier's weapon. As a result, the soldier can point his or her gun around a corner to assess enemy activity while minimizing exposure to attack.
A Land Warrior's head-mounted display serves as a targeting system. A soldier aims his or her weapon by placing a crosshair over an enemy unit in the scope video. In the heat of battle, however, it is difficult to quickly determine whether an individual is a friendly or enemy unit. As a result, friendly fire occurs in which a soldier accidentally fires on a fellow soldier.
To combat friendly fire, the present inventor developed a system that differentiates friend from foe in a soldier's head-mounted display. The system leverages position information reported by friendly soldiers. In addition, if aerial imagery of the battlefield is available, the system enables commanders to tag unknown individuals as friend, foe, or neutral.
Existing systems served as a foundation for the present invention. These existing systems are described below.
Tracking Objects in Video
The present invention relies on an object tracker to detect units in aerial imagery and scope video. Object trackers perform image analysis to locate, recognize, and follow objects in video sequences. Inventors have devised several suitable object trackers. For example:
U.S. Pat. No. 6,529,613 to Astle (2003) relies on image templates to determine the positions of objects in video.
U.S. Pat. No. 6,507,660 to Wirtz et al. (2003) uses a reference image and performs edge detection to identify objects in aerial imagery.
U.S. Pat. No. 6,542,621 to Brill et al. (2003) performs probabilistic analysis to deal with occlusion while tracking multiple objects in video sequences.
U.S. Pat. No. 6,434,254 to Wixson (2002) detects objects in video images using one method suitable for daytime scenes and a second method suitable for nighttime scenes.
The present invention builds on these object trackers by associating objects detected in aerial imagery with those detected in scope video. As a result, when a commander identifies an enemy unit in the aerial imagery, the present invention labels the corresponding unit in a soldier's scope video.
Determining an Object's Position
The present invention determines the position of unknown units in aerial imagery and uses this information to identify unknown units that appear in a soldier's scope video. Several existing systems use multiple cameras to determine the position of objects. For example:
U.S. Pat. No. 6,539,330 to Wakashiro (2003) determines the three-dimensional position of objects based on images from two cameras.
The present invention takes a different approach in determining the position of unknown units. The battlefield's topology restricts one dimension of movement for ground troops. For the remaining two dimensions, the present invention performs a calculation based on the unknown unit's position in an aerial image and the image's latitude and longitude extents. This technique enables the present invention to deduce the position of an unknown unit using a single camera.
Labeling Objects in Video
The present invention labels friendly and enemy units that appear in a soldier's head-mounted display. Similarly, several existing systems identify objects such as waypoints that appear in real-world images. For example:
U.S. Pat. No. 5,786,849 to Lynde (1998) identifies waypoints, obstacles, and targets that appear in marine navigation binoculars.
The present invention combines information about the position and orientation of a soldier's scope video camera with knowledge friendly and enemy unit locations to accurately label objects in a soldier's head-mounted display.
Differentiating Friend from Foe
In response to friendly fire incidents during the Gulf War, the U.S. Army launched a combat identification program. The program's goal was to equip soldiers with systems for differentiating friend from foe in combat. The result was an interrogator/transponder pair carried by each friendly unit. To determine whether a unit was a friend or foe, a soldier aimed his or her interrogator at the unit. For units carrying a properly configured transponder, the interrogator would indicate that the individual was a friend. Otherwise, the system deduced that the unit was an enemy.
The present invention has several advantages over combat identification systems that employ an interrogator and transponder. First, the present invention requires no equipment in addition to that already worn by Land Warriors. Second, the present invention requires to effort on the part of soldiers to differentiate friend from foe. The present invention automatically identifies units that appear in a soldier's scope video. As a result, the present invention enables soldiers to make rapid firing decisions.
Prior Art Disadvantages
Existing systems for differentiating friend from foe in combat suffer from a number of disadvantages. Specifically, existing systems:
a. Delay firing decisions. Existing systems require that a soldier obtain line-of-sight access and initiate an identification sequence before determining whether a unit is a friend or foe. As a result, soldiers need to wait for units to come into view and to manually perform an interrogation before making a firing decision.
b. Fail to eliminate friendly fire. A soldier who improperly aims his or her interrogator or fires in the heat of battle before initiating an identification sequence may injure or kill fellow soldiers.
c. Fail to identify unknown units as friend, foe, or neutral. Although existing systems differentiate friendly units from unknown units, they cannot further classify unknown units as friend, foe, or neutral. As a result, soldiers must visually confirm at close range an enemy's identity.
d. Fail to provide advance warning when an enemy is approaching. Existing systems do not identify distant or obstructed units. As a result, soldiers are susceptible to surprise attack as enemy units suddenly come into view.
e. Require line-of-sight. To identify a unit using an existing system, a soldier must see his or her adversary. Unfortunately, a visible enemy unit may launch an attack before a friendly unit can initiate the identification sequence. As a result, existing systems place soldiers at risk of attack while they differentiate units.
f. Fail to expose hidden enemies. Existing systems require that a soldier obtain a clear view of an enemy before initiating the identification sequence. As a result, existing systems leave soldiers vulnerable to attack by obscured or distant enemies.
g. Require that soldiers carry interrogators and transponders. Land Warriors carry 90 pounds of equipment. The heavy load limits a soldier's range and maneuverability. Therefore, the U.S. Army would like to eliminate interrogators and transponders from combat identification systems.